A marketing executive walked into her organization’s IT group and told the first person she saw that she needed help. All she wanted, she said, was to export her WordPress site’s subscriber data into an Excel spreadsheet. Simple.
That task wasn’t simple at all, not with the scripts someone in her business unit had his young son install. The IT group’s manager tells the story and adds in a surprising burst, “I hate shadow IT!”
Business units resort to shadow IT when their organization’s central IT departments say there’s no budget, no time, and, frankly, no interest. “Wait,” says IT.
But these desperate units can’t wait—they’ve got businesses to run. So semi-skilled techs within the group do what they can, which is plenty, thanks to the wave of ever easier-to-use solutions.
That might be fine, except for the inevitable disruption, such as the marketing executive’s WordPress site. Even worse, it exposes entire organizations to ransomware’s hot breath.
What could possibly resolve this dilemma?
A slew of academic research has pondered shadow IT, and most researchers seem to come out of the “I hate shadow IT!” corner. Judging from their tone, they view shadow IT systems like a basement nest of rats. Solutions amount to IT’s equivalents of poison, traps, and predators.
A friendlier tone came from a recent Gartner Inc. webinar, “CIO Strategy: Manage the Opportunities & Risks of Democratized Business-Led IT.” One slide read, “CIOs can no longer ‘own’ the entire organizational technology estate.” Correct. “And there is danger should they try to!” Agreed. “Half of business unit leaders with technical capacity think they have a better understanding of requirements they need than the IT organization.” Probably so, and perhaps for good reason.
One 30-year veteran of data management consulting comes out from shadow IT’s corner swinging hard. “How can you be angry at shadow IT?” said Evan Levy, partner with IntegralData, and more recently of SAS Analytics. For more than 30 years, he’s been one of the IT industry’s most sought after data management consultants.
“You can be angry at people breaking certain types of data guidelines,” he said, “but that’s probably because you haven’t invested in governance. You haven’t invested in frameworks for people to be more self-sufficient.”
The crux of the problem, he said, is that IT departments have often failed to create new business models that respond to evolving needs. IT is usually so focused on the old model that they’re not supporting the business.
Business units are justified in telling IT, “I’ve got a business responsibility to address. I’m not here to conform to your standards. You need to conform to mine!”
In Levy’s view, IT has always been in the role of putting itself out of business. Its original mandate, he said, was to reduce costs by automating business processes. It succeeded. But in the decades since, things evolved away from mass automation and into custom business process enablement. So, while IT still runs the big automation systems, newer technology and the business needs they serve have outrun IT.
Central IT isn’t even structured to serve those newer needs, he said. It’s typically managed by budgets, not customer initiatives.
“IT opened up Pandora’s box when they couldn’t respond to the needs that users had,” Evan explained. They can’t do a one-off report or load a one-off data source for a one-off user.
One particular peril of the shadow IT vs. central IT struggle is the rising chance of weakened security. Business unit techs are often naive about it. The IT department is, of course, shocked. But sometimes even IT departments themselves fail.
At least one veteran consultant who straddles the shadows and central IT’s bright lights finds that both sides in many organizations need improvement.
Dan Murray, director of strategic innovation at tech consultancy InterWorks, said that old-school thinking is still pervasive. The IT group, especially in midsize organizations, may have three people, and they’re all the kinds of guys who could fix the printer 10 years ago. “You’ve got to stay up with tech. It moves.”
Even worse, some central IT departments fail to appreciate security’s urgency. Among the common myths he hears is that the company is too small to be a target.
“You think you’re a small backwater?” said Murray. “You’re the ideal target. If I was in that business, I’d be going after you first because you’d be easy pickings, and you’d probably pay. And we’d be sensitive to how much you could afford.”
If central IT is close to outliving its era, and business units increasingly find themselves forced to serve themselves, the happy resolution points away from any central IT group and away from shadow IT. Instead, IT requirements might be met best by a group of consultants who can help as needed.
The IT manager who “hates shadow IT” and the marketing executive with the WordPress site are far from unique. Everyone seems to hate shadow IT. But each side hates it for different reasons. Without it, business units resort to do-it-yourself solutions outside of their expertise.
With it, central IT has a harder time producing one-size-fits-all solutions.
The best scenario is a transformation of IT into a service bureau. Another solid option is to hire an agency, similar to what Murray describes. Whichever way a company decides to go, the key is to shine the light on all shadow IT in your organization, and remove every trace of it.
The global big data analytics market is expected to reach $68.09 billion in annual revenue by 2025. It has become one of the most compelling technology trends reshaping business processes and operations. The category is now a crucial resource for both the public and private sectors, as well as for the healthcare industry.
Organizations can now track and analyze business data in real time, thanks to cloud-based applications. This allows for quicker turnaround time for adjustments. As we move into 2023, now is the time to anticipate, adapt, and scale your approach to big data by preparing for the coming year’s top trends in data and analytics.
Digital transformation pushes technology from every direction as we move deeper into the Internet of Things (IoT), machine learning, and big data. With IoT devices expected to move from 9.7 billion in 2020 to 29.4 billion in 2030, it’s easy to see how big data will play a pivotal role.
Artificial intelligence (AI) will become a bigger part of processing, trying to make sense of the vast amounts of data we’ve accumulated in the years past. We’ll continue to find applications for projecting insights on how we can grow in the future.
Business intelligence will continue moving into every industry and affect both strategic and tactical business decisions. Business intelligence (BI) leverages software and services to provide actionable insight to give users detailed intelligence about the state of the business.
With the value of the global BI and analytics software market expected to grow to $18 billion by 2025, growth may come in unexpected places. Collaborative BI can help unearth insights and make decisions without moving from a BI platform. It can make data more accessible and provide non-technical users an easy way to connect to the results.
More than 70% of companies have migrated at least some of their workloads to the public cloud, with even greater cloud adoption predicted in the next few years. A McKinsey survey found that companies averaged 23% over budget on cloud spending, and estimated that 30% of their outlays were wasted.
Cloud-native technologies are the only sustainable solution for our ever-changing business and IT infrastructure. They’re complicated, though, requiring standardized and self-service tools that allow even more non-technical users to better collect, analyze, and interpret usage and data.
Data as a Service (DaaS) has reached a point where even the smallest players can easily get into the game and generate revenue from it. With growth predicted to approach $11 billion in revenue by 2023, it’s expected to allow extreme niches to find value in their data.
If you have a company whose data could be of value to others, this could become a nice revenue stream. Now is the time to rethink what you’ve offered in the past, and find new ways to be relevant in the coming world.
More businesses are taking an active role in health and wellness. Examples include medical practices that are finding new ways to communicate with patients, and companies’ efforts toward identifying strategies to help keep workers healthy.
This trend can lessen the burden on our current medical system. The U.S. National Health Expenditure grew to $4.1 trillion in 2020, or $12,530 per person, accounting for 19.7% of GDP. By continuing to accumulate medical data from all across the planet, companies can use that data in new ways to identify cures sooner and faster than ever before.
While autonomous driving has been talked about for years, we’re finally getting closer to making it a reality. Waymo, Alphabet’s autonomous vehicle unit, now uses self-driving ride-hailing services in both its Phoenix and San Francisco locations. Walmart has used an autonomous vehicle program in Arkansas since 2020, and has registered more 70,000 miles.
As businesses continue using data derived from these early adapters, the only place to go is up. Big data will continue to help further define commuter transportation management programs, which should help shed light on how we’ll continue to move in the future.
Data analytics is the heart of all future development.
As businesses race to keep pace with the significant developments in digital transformation, one thing is clear: business intelligence will play a key role in future enterprise growth.
Businesses, big and small, that harness new technologies as they enter the market, utilizing essential self-service analytics tools and big data, will be better equipped to predict trends, identify business growth opportunities, and enhance profitability.
A data lake is a centralized repository where you store structured or unstructured data without having to transform it first. In other words, you simply store it as-is and use it for data analysis. You can store any type or volume of data in full fidelity, whether the data comes from on-premises, cloud, or edge-computing systems. This provides tremendous flexibility in storage on a scalable and secure platform.
A data lake allows you to process data in real-time or batch mode and analyze that data using SQL, Python, R, or any other language, third-party data, or analytics application.
There are several key reasons organizations choose data lakes.
The cost to store data in a data lake system is typically significantly lower than databases, which often require complex infrastructure and filtering. With a data lake, you only pay for the storage space. For organizations that store mass amounts of data, this can reduce the total cost of ownership (TCO).
Companies don’t need specialized hardware for a data lake, and data sets don’t need to be indexed and prepped before storage.
When organizations deploy data lakes, they can eliminate the data siloes that often exist. With a central repository, companies can also avoid the challenge of having to move data between data warehouses and data centers.
Used as a separate layer, a data lake can mitigate the costly egress fees – fees paid to transmit data out of a cloud platform – required to move data from one cloud provider to another for processing.
Traditional data warehouse platforms are schema-based, meaning that your data has to be stored in a specific format. When you set up your database, you have to decide its structure. Before data is added, it has to be cleansed and standardized. If new types or data formats are added, you may have to rebuild the database to accommodate it. With a data lake, you have flexibility because you can store any type or format of raw data.
Data that doesn’t fit squarely into a database slot, such as social media posts, customer support notes, images, sensor data, and other unstructured data, can still be available for data scientists.
A data lake makes data available across an enterprise to any authorized user. This allows companies to build a more data-centric culture and provide middle managers and frontline employees the data access they need to make better decisions.
Data warehouses generally rely on SQL, which may be fine for simple analytics. However, more advanced data analytics may require more flexibility in assessing data. Data lakes allow for more options. For example, you can store data from multiple sources in multiple forms as raw data for data scientists.
Data is then prepared for applications such as artificial intelligence and machine learning, predictive analytics, and data mining.
While there are plenty of benefits, challenges also exist regarding data lakes that organizations need to overcome.
Data lakes make it easy to save everything. While storage is more cost-efficient in a data lake, it’s still an expense. Further, constantly evolving and generally complex legislation around the world is adding new challenges associated with storing certain kinds of potentially sensitive data. Organizations need to put parameters on what should be saved. This requires data governance policies.
While data doesn’t require cleaning or transforming to be stored in a data lake, an organization still needs strong governance to ensure data quality. This includes:
Data governance covers every aspect of storing and securing enterprise assets to ensure quality and accountability.
The list of major data providers includes the major players in the cloud service provider (CSP) space, including:
As you can see, data lakes have significant advantages in modern environments. But knowing their strengths and weaknesses before investing a dime is crucial to getting the most out of them. Make sure they’re a good fit for your environment and business objectives first.
