Oops (Generated with ChatGPT) / RCBJ
Some organizations keep following the same legacy, bad practices even though they know better. They keep doing it right up until the unfortunate happens.
Yes, the title was designed to get you to click on this article…it worked! The title is relevant to this post’s content.
As always, please note, this isn’t about a specific organization. Some of the scenarios I mention below I read in other articles, heard in conversations, or encountered elsewhere.
In a previous life, before I was focused on Identity and Access Management (IAM) and application security, I was in the integration space. Fifteen, plus, years ago, this was Service Oriented Architecture (SOA) and Enterprise Service Buses (ESBs); ten years ago, this was API Management and some type of persistent queuing mechanism; seven years ago, it was Kafka and probably API Management. All of these tend to be around in some form to this day. My career originally drifted into identity when I was tasked with figuring out how integrate the identity stack with the ESB I was building 2010. Yes, it was all SOAP Web Services — let’s move on. The answer at the time was WS-Trust and SAML2. Most integration technologies I’d worked with up to that point didn’t really play well with the enterprise IAM stack, if there was even an identity (authentication + authorization) solution built into the integration layer of the architecture. There certainly wasn’t a standards-based approach to this identity integration. Okay, maybe LDAP, but that isn’t really in the spirit of what we were trying to do.
That project was the first time I ever worked with stateless bearer tokens attached to individual service requests as a secure, user-security-context propagation mechanism. This concept has been attached to every application identity solution I’ve deployed since then. But, that isn’t what we’re going to talk about today. We can explore the evolution of “stateless bearer tokens” and my career transition from the integration space to security over the past fifteen years in a future post.
There are moments that I miss being focused on the software integration space; there’s still a lot of room in the industry to get that right. This post is going to focus on a profoundly bad practice that I encounter occasionally in the integration space. The pattern in question is using web application screen scraping as an integration pattern instead of APIs, SOAP Web Services, cross-organization enterprise messaging (Websphere MQ), streaming messaging services (Kafka-like, or as I like to say, Kafka-esque), proprietary integration systems (think the integration stacks that the largest players offer), or other common integration patterns/technologies — though, these days, I see more REST APIs and Kafka then any of the others, but the rest of it is still still out there in different quarters. I want to explore the implications of using this pattern.
At this moment, there is a subset of architects and developers out there reading this, who are shaking their head wondering, what’s the big deal, we do this all over the place in my team or organization. Please, keep reading.
So, what can go wrong with using Web Application Screen Scraping (WASS) as an integration pattern?
Consider that it:
- Creates a counterintuitive contract between HTML UI element IDs and the screen scraping data consumers.
- Adds unplanned load on application servers, databases, and supporting systems that were not intended or planned for. The groups that start doing WASS for integration are not usually the groups that created the web application.
- Mixes online and batch/system/automated traffic.
- Uses identity patterns for the web application that were not designed for headless automated integration systems. Those headless libraries being used to navigate complex authentication flows tend to be fragile — very fragile.
- Tends to be used by business users or non-technical IT resources that will have very bad security practices — especially concerning passwords and PII data.
- Falls outside of mainstream IT practices and governance, which tends to create situations like each team buying their own tooling, which tends to create duplicated IT capabilities that are riddled with bad practices and increased costs.
Why would someone do this? There’s a couple of possible reasons I’ve encountered.
- Almost anything I can list here comes down to either no integration solution was given or an inadequate solution was given by the service / data provider.
- No API available.
- Lack of functionality in an API, but the web application UI provides the needed data or capabilities.
- The data consumer is unable to support the identity model required by the data provider.
- Lack of connectivity. Maybe the API is only accessible over a private networking link.
- Politics. Every organization has someone channeling their inner Gandalf-commanding-“You-Shall-Not-Pass”-to-the-balrog towards groups that are attempting to satisfy business requirements.
- Laziness. Either the data / service provider or data / service consumer doesn’t want to put in the effort to provide an API or call the available API.
- Money. Either the data / service provider or data / service consumer doesn’t want to spend the money to provide an API or update their systems to call the available API.
Regardless of the reasoning, at the end of the day, what’s the difference between web app screen scraping and third-party bots? Intentions and knowledge of the third-party, I suppose. AI bots scraping websites for content has been a major problem lately for websites operators from an availability and cost perspective. These are variations on the same basic problem for the data / service provider. Again, the motivation for the data / service consumer can vary dramatically. It’s important to understand why your data consumer’s are doing this to your website / web application.
On the data / service consumer side, if you are doing this to your vendors, business partners, or teams within your organization, I’d encourage you to stop doing it and either start using the REST APIs that are offered or encourage your data / service provider to provide an adequate solution. I’ve often seen that it isn’t the IT department on the data / service consumer side doing this, but rather business users attempting to compensate for functionality that they don’t have access to. So, you now have users outside of IT deploying a technical solution without understanding the implications. They probably downloaded, borrowed, or bought some type of QA automation solution that can record browser activity, parameterize a couple of query parameters or form inputs, and run it on their desktop to get their daily work done quickly. Maybe the business users don’t have adequate access on their device to actually install / run unsanctioned software; so, what is to be done, but setup the automation (if you can call it that) software from a personal device to make their jobs easier. Maybe it’s done from home; maybe the device is brought into the office. Regardless, now you have your organization’s (or customer’s or business partner’s) data outside of your organization’s control, you possibly have an unmanaged device on your network, and the situation has drifted outside of any threat model that has ever been considered by information security.
Meanwhile, the business user probably didn’t have the skill sets to automate the user authentication process; especially, if MFA is used. So, the user has to, first, manually log into the browser session that the automation tool is managing. Or, better yet, maybe they did manage to automate the authentication process and multiple userids are used to authenticate to complete some type of workflow process. Or, even better yet, maybe every business user in the group has their part to play in each automated workflow. So, they have every user’s credentials written down somewhere. Where to keep all those credentials? It’s always a spreadsheet…on a shared drive…probably with world-readable permissions.
So, right there, we have:
- Credentials are being improperly stored in clear text rather than in the approved secure credentials storage solution.
- Everybody can access the credentials.
- All audit logging on the web application is meaningless because you have no idea who is actually running the automation that day. Meanwhile, the data / service consumer organization’s IT department probably has some type of asset management governance capability that is supposedly tracking all the software solutions in use in the organization. What are the chances the business user group bothered notifying asset management of what tools they are using? Or, that they even know something like that should be done?
- Is the automation software properly licensed and paid for? How does it get renewed each year?
- Any Threat Modeling performed around the business user’s accessing this web application (for consumer or provider organization) is probably missing a critical use case.
- How many different teams in the data / service consumer organization have done this outside of normal IT processes? How many duplicated phantom software packages have been bought to solve the underlying capability gap? The record I’ve seen is three.
Hopefully, you see that WASS can lead to all kinds of problems in the information security and IT governance spaces.
Maybe, all of these issues are addressed within your organization and none of these issues are occurring. That’s nice, but you are probably still causing significant issues on the data / service provider side. Of course, that doesn’t impact you or your budget until their lawyers come knocking at your door.
The whole politics, funding, laziness trifecta is harder to address — sometimes bordering on impossible. If it is business users engaging in this integration outside of mainstream IT, then I would encourage the IT department to understand the gaps in services / capabilities being providing to their users.
If you are on the data / service provider side of this issue and you haven’t been able to get your customers to stop doing this to your systems, you could build separate infrastructure for the screen scraping traffic to isolate the impact to regular users. Of course, this assumes that you are even aware that this is happening to your systems. Identify which users identities are being used for WASS and direct that traffic to the dedicated infrastructure. You could ask them nicely to use a new URL (good luck with that), maybe a load balancer (F5 BigIP, etc) could identify these users and redirect to the appropriate backend servers, maybe the same load balancer could identify the automated traffic by on the User-Agent HTTP Request Header (this is easy to fake with headless browser libraries). This may include web servers, app servers, database servers, CICS regions, etc. Admittedly, this is an expensive solution.
If you are the data / service provider and can fill the gap, maybe the offending data / service consumer won’t need to hammer your web application so hard. I’m speaking primarily from the standpoint of corporate business web applications at this point. Fending off third-party AI bots that are trying to suck in all your content for training their LLMs is a more complex problem.
If you have addressed gaps and provided a fully-functional API layer, but the data / service consumers continue to use the WASS pattern, then it’s time to discuss increases in fees, getting corporate legal involved, throttling their accounts, simply disabling their access, or some of the other deterrence options described below. Of course, these options assume that you can handle a disruption in the related revenue because it’s unlikely they will immediately and enthusiastically agree to paying more.
Please note, I said “web application screen scraping”, not (mainframe) green-screen screen scrapping. The latter always struck me as a bit lazy as well, but it’s easy enough to create a dedicated CICS region to handle green-screen screen scrapping — it has been done many times with successful results. That being said, you could also deploy a much more elegant solution such as exposing the mainframe database (DB2, Oracle) listener and calling directly via an appropriate driver within a services layer, using Broadcom HostBridge to invoke CICS transactions directly from that services layer, using Oracle Tuxedo Mainframe Adapter to invoke CICS transactions from the services layer (this was my first job out of school), or a couple of other options in that space that we don’t need to go into here.
So, what can be done by a data / service provider to prevent this from happening? Here are some patterns to deter the WASS integration anti-pattern:
- rate-limiting
- identifying automated traffic (by source IP, User-Agent Request Header, user identity) and blocking (or taking other actions)
- identifying known malicious entities and traffic associated with these entities and blocking (or taking other actions)
- honey-potting
- Provide easy-to-use APIs or other integration patterns.
- randomly generate UI element IDs.
- Use an IdP decoupled from the application that utilizes anti-bot capabilities (approaches vary).
- MFA for all users — this could still be automated, but is much harder to accomplish.
- Do not use service accounts; all user accounts should map to a human user.
- robots.txt — obviously, bad actors are not going to obey this, but good to have the basics covered.
- Request headers validation
- CAPTCHAs and JavaScript challenges — this is getting easier to bypass with some of the modern AI tools.
- Browser and user fingerprinting
- geolocation blocking
- Web Application Firewalls (WAFs)
None of these are guaranteed to address the issue, but if you use libraries that incorporate these capabilities and / or deploy the capabilities using appropriate services, you will likely make it much harder for organizations to WASS your web applications.
What Should Be Done?
Implement one of the following:
- APIs
- Message Streaming
- Traditional Messaging
The Web Application front end should incorporate the strategies listed above.
Keep communication lines with your data / service provider customers to address whatever issues they are facing.
As noted earlier, there are libraries that implement many of the strategies for making it harder to screen scrap your web application. There are also client-side libraries that seek to avoid being detected by these strategies. In some cases, the libraries that serve these opposing goals are written by the same Open Source Software projects — the OSS community is now selling weapons to both sides. We go on.
Every organizations situation is different. Maybe the data / service provider and consumer organizations are happy with this approach.
Notes:
- Photo: Generated by ChatGPT.
- AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
- Names have been changed to protect the guilty.
- None of the hostname or users used in examples actually exist.
- Feel free to post any comments or suggestions below.