• 3AG GUIDE TO

    DATA ENGINEERING

     
     
    In this comprehensive guide, we look at all
    aspects of data engineering to understand
    how this fundamental discipline is the
    foundation for all things data.

  • Introduction to

    Data Engineering

     
     

    So what is data engineering? And why does it matter?

    Perhaps this is best explained with a metaphor. Imagine that it’s 1970, and you’ve just opened a bookstore. For the grand opening you stock your store with the 10 most popular books from the New York Times bestseller list, but soon realize that customers are looking for a variety of books to purchase in addition to the most popular books. So you add books from the top-20, then the top-100.

     

    But now customers get confused when they enter the store, because you’ve gone from stacking books in 10 piles to stacking them in 20 and then 100 different piles. Worse yet, customers start mixing the different piles, which makes it even harder for them to find what they want.

     

    So you add a system of bookshelves, putting the books in order of popularity. Customers entering the store can now walk past the most popular books and make their way to the slowest sellers.

     

    But then certain customers ask you to make special orders.

    You decide that it makes sense to order a few extra since it’s a sure sign of demand.

     

    See a pattern here? This constant improvement and iteration of the organization of this bookstore’s data has strong parallels to the discipline of data engineering.

     

    Data engineers are charged with the mission of helping their company to organize its data in such a way as to be not only accessible to the widest audience, but useful for their particular needs. Some employees just need to look at a chart. Others need to do comprehensive analysis. And other employees need to structure the data to allow for advanced use cases such as machine learning and artificial intelligence.

     

    Data engineering is the discipline devoted to making all this possible, while simultaneously dealing with an ever-changing landscape, both internally and externally.

    Learn More

  • Why bother with

    Data Engineering?

     
     

    Again a metaphor will help. 

    Imagine your company as being either a shed, a house, a multi-family complex or a high-rise tower. The owner of a shed probably doesn’t spend much time concerned about engineering permits or maintenance since this person also doesn’t need to worry about maintaining services like heating, running water or electricity. And if the shed isn’t going to be upgraded anytime soon, there is little need to plan for future changes.

     

    The owner of a house will need to devote significantly more effort to keeping things running when compared with a shed, but even then they won’t need to concern themselves with the needs of themselves and their family.

     

    A multi-family complex requires more planning, more resource integration, and a more structured approach to dealing with issues as they arise, particularly since multiple tenants will live in the unit.

     

    And a high-rise tower will have a completely different set of engineering requirements, permitting requirements, safety requirements, and so on.

     

    And a high-rise tower will have a completely different set of engineering requirements, permitting requirements, safety requirements, and so on. The high-rise tower will also be much more complicated to manage, particularly if things like fixtures have not been standardized across different units in the building.

     

    See a trend here? The more complicated the structure, the greater the need for better infrastructure. More subtle is the fact that earlier planning becomes even more important for bigger buildings, as mistakes and omissions can become costly later on.

     

    The same holds true for data engineering within a company. While the smallest companies may not have an immediate need for tools and structures to support data access, companies with aspirations to become bigger will need to consider this issue at some point in their growth. And the longer they wait, the more expensive the effort will be, with more work required to fix past mistakes and omissions.

    Learn More

  • Developing a

    Data Strategy

    Without a clear data strategy, companies run the risk of carving off small data projects (or worse yet no projects at all) that don’t connect, resulting in a sum that is less than the parts.

    Forward thinking companies understand the value of knowing where they are, and where they want to go. From a data perspective, this assessment involves understanding where they fit in the analytics maturity model.

    The majority of companies that have not formally developed a data strategy tend to fall in the descriptive category, performing basic reporting. Those that make a conscious decision to focus on improving their reporting will advance to the diagnostic phase of their journey, where they not only centralize data

    to improve the efficiency of their reporting, but also examine the question

    of “why” when doing analysis. More advanced stages of maturity include predictive and prescriptive, but in reality few companies achieve this level

    of expertise across the entire organization.

     

    One common mistake made by many companies is to jump into advanced stage use cases without getting the basics right, or even knowing where

    they are in terms of analytics maturity.

    Doing so not only potentially ignores lower-hanging opportunities for improvement, but in the worst case may set the company up for failure if it chooses to take on a project that ignores earlier stage prerequisites. For example, a company looking to do predictive analysis to support production planning will not be able to efficiently or reliably perform analysis if all the source data is spread across hundreds of spreadsheets managed by dozens of people.

     

    At 3AG, we developed the Data Coach with the specific purpose of helping companies to assess their current analytics maturity level, their target, and the necessary steps required to achieve this goal. Other organizations also offer similar roadmap development services – the important this to consider is that you have a clear plan of attack.

    Learn More

  • Data Governance

    Data governance is where your company gets to set it policies related to

    how data is collected. It is tightly integrated with data strategy in that

    it is the implementation of the rules defined in the former.

     

    A solid data governance policy will not only look to identify which sources to consolidate, but it will define who will be able to access data, when the data is pulled, and what do to when conflicts might occur. The governance policy also defines other issues such as retention policies and how to integrate new sources of data as they come online.

  • Wrapping Up

    We hope that you have found this resource helpful. Data engineering is a complex topic, made more so by the fact that there are countless ways for an organization to tackle it.

     

     

    For organizations looking to get more control over their data but who lack experience in this area, a guide can be invaluable. Consider bringing in experts who can give an unbiased assessment of your current situation, and of course we would be honored if you were to consider the 3AG Data Coach as an option.

     

All Posts
×
×
Terms & Conditions
Welcome to 3AG Systems
These terms and conditions outline the rules and regulations for the use of 3AG Systems's Website.  

3AG Systems is located at: 
100 - 3605 Gilmore Way, Burnaby BC  V5G 4X5, Canada

By accessing this website we assume you accept these terms and conditions in full. Do not continue to use 3AG Systems's website if you do not accept all of the terms and conditions stated on this page.

The following terminology applies to these Terms and Conditions, Privacy Statement and Disclaimer Notice and any or all Agreements: “Client”, “You” and “Your” refers to you, the person accessing this website and accepting the Company’s terms and conditions. “The Company”, “Ourselves”, “We”, “Our” and “Us”, refers to our Company. “Party”, “Parties”, or “Us”, refers to both the Client and ourselves, or either the Client or ourselves. All terms refer to the offer, acceptance and consideration of payment necessary to undertake the process of our assistance to the Client in the most appropriate manner, whether by formal meetings of a fixed duration, or any other means, for the express purpose of meeting the Client’s needs in respect of provision of the Company’s stated services/products, in accordance with and subject to, prevailing law of Canada. Any use of the above terminology or other words in the singular, plural, capitalisation and/or he/she or they, are taken as interchangeable and therefore as referring to same.

Cookies
We employ the use of cookies. By using 3AG Systems's website you consent to the use of cookies in accordance with 3AG Systems’s privacy policy.Most of the modern day interactive web sites use cookies to enable us to retrieve user details for each visit. Cookies are used in some areas of our site to enable the functionality of this area and ease of use for those people visiting. Some of our  affiliate / advertising partners may also use cookies.

License
Unless otherwise stated, 3AG Systems and/or it’s licensors own the intellectual property rights for all material on 3AG Systems. All intellectual property rights are reserved. You may view and/or print pages from https://www.3agsystems.com for your own personal use subject to restrictions set in these terms and conditions.

You must not:
Republish material from https://www.3agsystems.com
Sell, rent or sub-license material from https://www.3agsystems.com
Reproduce, duplicate or copy material from https://www.3agsystems.com
Redistribute content from 3AG Systems (unless content is specifically made for redistribution).


Hyperlinking to our Content
The following organizations may link to our Web site without prior written approval:
Government agencies;
Search engines;
News organizations;
Online directory distributors when they list us in the directory may link to our Web site in the same manner as they hyperlink to the Web sites of other listed businesses; and
Systemwide Accredited Businesses except soliciting non-profit organizations, charity shopping malls, and charity fundraising groups which may not hyperlink to our Web site.

These organizations may link to our home page, to publications or to other Web site information so long as the link: (a) is not in any way misleading; (b) does not falsely imply sponsorship, endorsement or approval of the linking party and its products or services; and (c) fits within the context of the linking party's site.

We may consider and approve in our sole discretion other link requests from the following types of organizations:
commonly-known consumer and/or business information sources such as Chambers of Commerce, American Automobile Association, AARP and Consumers Union;
dot.com community sites;
associations or other groups representing charities, including charity giving sites,
online directory distributors;
internet portals;
accounting, law and consulting firms whose primary clients are businesses; and
educational institutions and trade associations.


We will approve link requests from these organizations if we determine that: (a) the link would not reflect unfavorably on us or our accredited businesses (for example, trade associations or other organizations representing inherently suspect types of business, such as work-at-home opportunities, shall not be allowed to link); (b)the organization does not have an unsatisfactory record with us; (c) the benefit to us from the visibility associated with the hyperlink outweighs the absence of ; and (d) where the link is in the context of general resource information or is otherwise consistent with editorial content in a newsletter or similar product furthering the mission of the organization.

These organizations may link to our home page, to publications or to other Web site information so long as the link: (a) is not in any way misleading; (b) does not falsely imply sponsorship, endorsement or approval of the linking party and it products or services; and (c) fits within the context of the linking party's site.

If you are among the organizations listed in paragraph 2 above and are interested in linking to our website, you must notify us by sending an e-mail to info@3agsystems.com.

Please include your name, your organization name, contact information (such as a phone number and/or e-mail address) as well as the URL of your site, a list of any URLs from which you intend to link to our Web site, and a list of the URL(s) on our site to which you would like to link. Allow 2-3 weeks for a response.

Approved organizations may hyperlink to our Web site as follows:
By use of our corporate name; or
By use of the uniform resource locator (Web address) being linked to; or
By use of any other description of our Web site or material being linked to that makes sense within the context and format of content on the linking party's site.

No use of 3AG Systems’s logo or other artwork will be allowed for linking absent a trademark license agreement.


Reservation of Rights
We reserve the right at any time and in its sole discretion to request that you remove all links or any particular link to our Web site. You agree to immediately remove all links to our Web site upon such request. We also reserve the right to amend these terms and conditions and its linking policy at any time. By continuing to link to our Web site, you agree to be bound to and abide by these linking terms and conditions.

Removal of links from our website
If you find any link on our Web site or any linked web site objectionable for any reason, you may contact us about this. We will consider requests to remove links but will have no obligation to do so or to respond directly to you.

Whilst we endeavour to ensure that the information on this website is correct, we do not warrant its completeness or accuracy; nor do we commit to ensuring that the website remains available or that the material on the website is kept up to date.


Content Liability
We shall have no responsibility or liability for any content appearing on your Web site. You agree to indemnify and defend us against all claims arising out of or based upon your Website. No link(s) may appear on any page on your Web site or within any context containing content or materials that may be interpreted as libelous, obscene or criminal, or which infringes, otherwise violates, or advocates the infringement or other violation of, any third party rights.


Disclaimer
To the maximum extent permitted by applicable law, we exclude all representations, warranties and conditions relating to our website and the use of this website (including, without limitation, any warranties implied by law in respect of satisfactory quality, fitness for purpose and/or the use of reasonable care and skill). Nothing in this disclaimer will:
limit or exclude our or your liability for death or personal injury resulting from negligence;
limit or exclude our or your liability for fraud or fraudulent misrepresentation;
limit any of our or your liabilities in any way that is not permitted under applicable law; or
exclude any of our or your liabilities that may not be excluded under applicable law.

The limitations and exclusions of liability set out in this Section and elsewhere in this disclaimer: (a) are subject to the preceding paragraph; and (b) govern all liabilities arising under the disclaimer or in relation to the subject matter of this disclaimer, including liabilities arising in contract, in tort (including negligence) and for breach of statutory duty.

To the extent that the website and the information and services on the website are provided free of charge, we will not be liable for any loss or damage of any nature.


Credit & Contact Information
If you have any queries regarding any of our terms, please contact us.
×
Privacy Policy
We collect personal information from our customers in the regular course of doing business. This brochure answers some of your most frequently asked questions, and lets you know exactly how we're protecting the information you entrust to us.


What personal information do you collect about me?

We collect the following information about you:
-Name
-Email Address
-Payment card number
-Payment card expiry date
-Purchase history
-Product/service preferences
-Customer satisfaction info
-Opinions about products and services

When you visit our web site, we also collect:
-information about your computer, including your IP address, the type of operating system and browser you use, and your computer's location
-what pages you visit on our site and what links you click on
-what other sites you've visited recently


How do you use this information?

The main reasons we collect personal information from you are:
-Customer service
-Marketing
-To complete a sale/transaction

If it's a necessary part of any of these transactions, we may disclose your information to another company. For example, when you apply for credit, we pass on your personal information to a credit reporting agency so we can complete a credit check. We also pass on your name and address to a courier company to complete a delivery.


Use of Personal Information for Secondary Reasons

We also may use your personal information for other, secondary reasons, including:

To complete a sale/transaction
-Name
-Email Address
-Payment card number
-Payment card expiry date
-Purchase history
-Product/service preferences

Marketing
-Name
-Email Address
-Purchase history
-Product/service preferences
-Customer satisfaction info
-Opinions about products and services

Customer service
-Name
-Email Address
-Purchase history
-Product/service preferences


Sharing of Personal Information with Third Parties

Sometimes, we also share your personal information with other companies, including:
-Partners

We will also disclose your personal information if we are required by law to do so.



How do you get my consent?

When you provide us with personal information to complete a transaction, verify your credit card, place an order, arrange for a delivery or return a purchase, we assume you consent to our collecting it and using it for that specific reason only.

If we ask you for personal information for a secondary reason, like marketing, we will either ask you directly for your consent or provide you with an opportunity to say no. Saying no is called "opting out". By opting out, you can tell us not to collect the information and/or not to share it with other companies.



How do I opt out?

Contact email us at privacy@3agsystems.com with the subject line "Opt out".



How do I get more information?

Our staff will be happy to answer any questions you may have about your personal information. If you would like more information about our policies, or you would like to see exactly what personal information we have about you in our records, or you wish to register a complaint, please contact:

Email: privacy@3agsystems.com

You can also contact the Privacy Commissioner of Canada for assistance between the hours of 8:30 a.m. to 4:30 p.m. est, at:

Toll-free: 1-800-282-1376
Phone: (819)994-5444
Fax: (819)994-5424
TTY: (819)994-6591

or by mail at:

30 Victoria Street
Gatineau, Quebec
K1A 1H3

or on the web at:

http://www.priv.gc.ca