THE 3AG GUIDE TO
DATA ENGINEERING
In this comprehensive guide, we look at keyaspects of data engineering, which togethercreate a solid foundation for all things data.
Introduction to
data engineering
What is data engineering? And why does it matter?
Perhaps this is best explained with a metaphor. Imagine it’s 1970, and you’ve just opened a bookstore. For the grand opening, you stock the 10 most popular books from the New York Times bestseller list, but soon realize customers are looking for other books in addition to the most popular ones. You add books from the top 20, then the top 100.
But now customers get confused when they enter the store, because you’ve gone from stacking books in 10 piles to stacking them in 20 and then 100 different piles. Worse yet, customers start mixing the different piles, which makes it even harder for them to find what they want.
So, you add a system of bookshelves, putting the books in order of popularity. Customers entering the store can now walk past the most popular books and make their way to the slowest sellers.
But then certain customers ask you to make special orders.
You decide that it makes sense to order a few extra since it’s a sure sign of demand.
See a pattern here? This constant improvement and iteration of the organization of bookstore data has strong parallels to the discipline of data engineering.
Data engineers are charged with helping their companies organize data to be not only accessible to the widest audience, but also to be useful for their particular needs. Some employees just need to look at a chart. Others need to do comprehensive analysis. And other employees need to structure data to allow for advanced use cases like machine learning and artificial intelligence.
Data engineering makes this possible, while also dealing with ever-changing internal and external business landscapes.
Learn more
Why consider
data engineering?
Another metaphor will help.
Imagine your company owns either a shed, a house, a multi-family complex, or a high-rise. The shed owner probably doesn’t spend much time thinking about engineering permits or maintenance since they don’t need to worry about services like heating, running water, or electricity. And if the shed won’t be upgraded anytime soon, there's no need to plan for future changes.
The house owner will need to devote significantly more effort to keeping things running; even so, they won’t need to concern themselves with more than their own and their family's needs.
A multi-family complex requires more planning, more resource integration, and a more structured approach to handling issues as they arise; and different tenants may have very different needs.
And a high-rise tower will have a completely different set of engineering requirements, permitting requirements, safety requirements, and so on.
The high-rise will also be much more complicated to manage, particularly if things like fixtures have not been standardized throughout the building.
See a trend here? The more complicated the structure, the greater the need for better infrastructure. And earlier planning becomes even more important for bigger buildings, as early mistakes and omissions can become costly later on.
The same holds true for data engineering within a company. While the smallest companies may not immediately need tools and structures to support data access, companies with growth aspirations will need to consider this issue. And the longer they wait, the more expensive the effort will be, with more work required to fix past errors and omissions.
Learn more
Developing a
data strategy
Without a clear data strategy, companies risk carving off small data projects (or no projects at all) that don’t connect, either to one another or to their central databases.
Forward-thinking companies see the value of knowing both where they are, and where they want to go, on their data journey. From a data perspective, this involves understanding where they fit into the analytics maturity model.
Most companies that haven't formally developed a data strategy tend to fall in the descriptive category, performing basic reporting. Those that consciously decide to focus on improving their reporting advance to the diagnostic phase, where they not only centralize data to improve reporting efficiency, but also ask “why” when doing analysis. More advanced stages of maturity include predictive and prescriptive, but few companies achieve this level of expertise across the entire organization.
Common mistakes many companies make include jumping into advanced use cases without getting the basics right, or starting to plan before knowing what analytics maturity stage they're at.
Skipping steps can mean missing lower-hanging opportunities for improvement. Such haste can set a company up for failure if begins projects not built on a solid foundation of earlier stage prerequisites.
For example, a company looking to do predictive analysis to support production planning won't be able to efficiently or reliably perform analysis if source data is spread across hundreds of spreadsheets managed by dozens of people.
The 3AG Data Coach was developed specifically to help companies assess their current analytics maturity level, their targets, and the steps required to succeed.
Learn more
Wrapping up
Data engineering is a complex topic, made more so by the countless ways organizations can tackle it.
For organizations looking to gain more control over their data but lack experience in this area, a guide is invaluable.
Consider bringing in experts who can give an unbiased assessment of your current situation. We would be honored if you consider 3AG's Data Coach as your best option.
Welcome to 3AG Systems These terms and conditions outline the rules and regulations for the use of 3AG Systems's Website. 3AG Systems is located at: 100 - 3605 Gilmore Way, Burnaby BC V5G 4X5, Canada By accessing this website we assume you accept these terms and conditions in full. Do not continue to use 3AG Systems's website if you do not accept all of the terms and conditions stated on this page. The following terminology applies to these Terms and Conditions, Privacy Statement and Disclaimer Notice and any or all Agreements: “Client”, “You” and “Your” refers to you, the person accessing this website and accepting the Company’s terms and conditions. “The Company”, “Ourselves”, “We”, “Our” and “Us”, refers to our Company. “Party”, “Parties”, or “Us”, refers to both the Client and ourselves, or either the Client or ourselves. All terms refer to the offer, acceptance and consideration of payment necessary to undertake the process of our assistance to the Client in the most appropriate manner, whether by formal meetings of a fixed duration, or any other means, for the express purpose of meeting the Client’s needs in respect of provision of the Company’s stated services/products, in accordance with and subject to, prevailing law of Canada. Any use of the above terminology or other words in the singular, plural, capitalisation and/or he/she or they, are taken as interchangeable and therefore as referring to same. Cookies We employ the use of cookies. By using 3AG Systems's website you consent to the use of cookies in accordance with 3AG Systems’s privacy policy.Most of the modern day interactive web sites use cookies to enable us to retrieve user details for each visit. Cookies are used in some areas of our site to enable the functionality of this area and ease of use for those people visiting. Some of our affiliate / advertising partners may also use cookies. License Unless otherwise stated, 3AG Systems and/or it’s licensors own the intellectual property rights for all material on 3AG Systems. All intellectual property rights are reserved. You may view and/or print pages from https://www.3agsystems.com for your own personal use subject to restrictions set in these terms and conditions. You must not: Republish material from https://www.3agsystems.com Sell, rent or sub-license material from https://www.3agsystems.com Reproduce, duplicate or copy material from https://www.3agsystems.com Redistribute content from 3AG Systems (unless content is specifically made for redistribution). Hyperlinking to our Content The following organizations may link to our Web site without prior written approval: Government agencies; Search engines; News organizations; Online directory distributors when they list us in the directory may link to our Web site in the same manner as they hyperlink to the Web sites of other listed businesses; and Systemwide Accredited Businesses except soliciting non-profit organizations, charity shopping malls, and charity fundraising groups which may not hyperlink to our Web site. These organizations may link to our home page, to publications or to other Web site information so long as the link: (a) is not in any way misleading; (b) does not falsely imply sponsorship, endorsement or approval of the linking party and its products or services; and (c) fits within the context of the linking party's site. We may consider and approve in our sole discretion other link requests from the following types of organizations: commonly-known consumer and/or business information sources such as Chambers of Commerce, American Automobile Association, AARP and Consumers Union; dot.com community sites; associations or other groups representing charities, including charity giving sites, online directory distributors; internet portals; accounting, law and consulting firms whose primary clients are businesses; and educational institutions and trade associations. We will approve link requests from these organizations if we determine that: (a) the link would not reflect unfavorably on us or our accredited businesses (for example, trade associations or other organizations representing inherently suspect types of business, such as work-at-home opportunities, shall not be allowed to link); (b)the organization does not have an unsatisfactory record with us; (c) the benefit to us from the visibility associated with the hyperlink outweighs the absence of ; and (d) where the link is in the context of general resource information or is otherwise consistent with editorial content in a newsletter or similar product furthering the mission of the organization. These organizations may link to our home page, to publications or to other Web site information so long as the link: (a) is not in any way misleading; (b) does not falsely imply sponsorship, endorsement or approval of the linking party and it products or services; and (c) fits within the context of the linking party's site. If you are among the organizations listed in paragraph 2 above and are interested in linking to our website, you must notify us by sending an e-mail to info@3agsystems.com. Please include your name, your organization name, contact information (such as a phone number and/or e-mail address) as well as the URL of your site, a list of any URLs from which you intend to link to our Web site, and a list of the URL(s) on our site to which you would like to link. Allow 2-3 weeks for a response. Approved organizations may hyperlink to our Web site as follows: By use of our corporate name; or By use of the uniform resource locator (Web address) being linked to; or By use of any other description of our Web site or material being linked to that makes sense within the context and format of content on the linking party's site. No use of 3AG Systems’s logo or other artwork will be allowed for linking absent a trademark license agreement. Reservation of Rights We reserve the right at any time and in its sole discretion to request that you remove all links or any particular link to our Web site. You agree to immediately remove all links to our Web site upon such request. We also reserve the right to amend these terms and conditions and its linking policy at any time. By continuing to link to our Web site, you agree to be bound to and abide by these linking terms and conditions. Removal of links from our website If you find any link on our Web site or any linked web site objectionable for any reason, you may contact us about this. We will consider requests to remove links but will have no obligation to do so or to respond directly to you. Whilst we endeavour to ensure that the information on this website is correct, we do not warrant its completeness or accuracy; nor do we commit to ensuring that the website remains available or that the material on the website is kept up to date. Content Liability We shall have no responsibility or liability for any content appearing on your Web site. You agree to indemnify and defend us against all claims arising out of or based upon your Website. No link(s) may appear on any page on your Web site or within any context containing content or materials that may be interpreted as libelous, obscene or criminal, or which infringes, otherwise violates, or advocates the infringement or other violation of, any third party rights. Disclaimer To the maximum extent permitted by applicable law, we exclude all representations, warranties and conditions relating to our website and the use of this website (including, without limitation, any warranties implied by law in respect of satisfactory quality, fitness for purpose and/or the use of reasonable care and skill). Nothing in this disclaimer will: limit or exclude our or your liability for death or personal injury resulting from negligence; limit or exclude our or your liability for fraud or fraudulent misrepresentation; limit any of our or your liabilities in any way that is not permitted under applicable law; or exclude any of our or your liabilities that may not be excluded under applicable law. The limitations and exclusions of liability set out in this Section and elsewhere in this disclaimer: (a) are subject to the preceding paragraph; and (b) govern all liabilities arising under the disclaimer or in relation to the subject matter of this disclaimer, including liabilities arising in contract, in tort (including negligence) and for breach of statutory duty. To the extent that the website and the information and services on the website are provided free of charge, we will not be liable for any loss or damage of any nature. Credit & Contact Information If you have any queries regarding any of our terms, please contact us.
We collect personal information from our customers in the regular course of doing business. This brochure answers some of your most frequently asked questions, and lets you know exactly how we're protecting the information you entrust to us. What personal information do you collect about me? We collect the following information about you: -Name -Email Address -Payment card number -Payment card expiry date -Purchase history -Product/service preferences -Customer satisfaction info -Opinions about products and services When you visit our web site, we also collect: -information about your computer, including your IP address, the type of operating system and browser you use, and your computer's location -what pages you visit on our site and what links you click on -what other sites you've visited recently How do you use this information? The main reasons we collect personal information from you are: -Customer service -Marketing -To complete a sale/transaction If it's a necessary part of any of these transactions, we may disclose your information to another company. For example, when you apply for credit, we pass on your personal information to a credit reporting agency so we can complete a credit check. We also pass on your name and address to a courier company to complete a delivery. Use of Personal Information for Secondary Reasons We also may use your personal information for other, secondary reasons, including: To complete a sale/transaction -Name -Email Address -Payment card number -Payment card expiry date -Purchase history -Product/service preferences Marketing -Name -Email Address -Purchase history -Product/service preferences -Customer satisfaction info -Opinions about products and services Customer service -Name -Email Address -Purchase history -Product/service preferences Sharing of Personal Information with Third Parties Sometimes, we also share your personal information with other companies, including: -Partners We will also disclose your personal information if we are required by law to do so. How do you get my consent? When you provide us with personal information to complete a transaction, verify your credit card, place an order, arrange for a delivery or return a purchase, we assume you consent to our collecting it and using it for that specific reason only. If we ask you for personal information for a secondary reason, like marketing, we will either ask you directly for your consent or provide you with an opportunity to say no. Saying no is called "opting out". By opting out, you can tell us not to collect the information and/or not to share it with other companies. How do I opt out? Contact email us at privacy@3agsystems.com with the subject line "Opt out". How do I get more information? Our staff will be happy to answer any questions you may have about your personal information. If you would like more information about our policies, or you would like to see exactly what personal information we have about you in our records, or you wish to register a complaint, please contact: Email: privacy@3agsystems.com You can also contact the Privacy Commissioner of Canada for assistance between the hours of 8:30 a.m. to 4:30 p.m. est, at: Toll-free: 1-800-282-1376 Phone: (819)994-5444 Fax: (819)994-5424 TTY: (819)994-6591 or by mail at: 30 Victoria Street Gatineau, Quebec K1A 1H3 or on the web at: http://www.priv.gc.ca