Unless You Really Want to Look Like a Fool, Don't Save on Software Testing

From time to time, I just can't resist the temptation to do some ad hoc testing of business web applications I run into on the Internet. I've been doing it for years off and on (no penetration or any other disruptive testing, of course), and, although I don't keep statistics, my subjective feeling is that web apps have gotten much buggier lately. What's even worse is that the nature of the bugs I come across these days makes me wonder whether whoever is in charge of those applications just slaps them together as quickly as possible and puts them in production without any testing at all.

If you think I am exaggerating, let me give you an example.



I found this one on a career site operated by a major department of a large U.S. state (it is not my intent to embarrass anybody, so the department, as well as the state, shall remain unnamed).



I don't need to tell you that, if a user cannot perform reasonably complex searches for jobs, a career site is pretty much useless. Naturally, that was the first thing I started messing with... and got some really curious results:

1. searching for "analyst" returns 187 rows (see screenshot 1 below this post);
2. searching for "landscaper" returns 9 rows (see screenshot 2 below this post);
3. searching for "analyst landscaper" (in any order, with or without quotes, with or without operators entered by user) returns 168 rows (see screenshot 3 below this post).

Odd, isn't it?

In case you didn't get it, let me explain:
Assuming that the searches using one keyword at a time work correctly (and, generally, such a simple thing is kind of difficult to screw up), there are 187 jobs containing the keyword "analyst" and 9 jobs containing the keyword "landscaper" in the database. It appears that the search ignores the AND and OR operators (as well as double or single quotes, parentheses, etc.), which means that it either uses a hardcoded default operator between the keywords or searches for the complete string entered into the "Keywords" text box of the search form.

Let's look at all possible scenarios:

1. if the search treats each keyword as a separate string and uses the OR operator, it should return between 187 (if all "landscaper" jobs also happen to contain the keyword "analyst") and 196 (if none of the jobs contains both keywords) rows;
2. if the search treats each keyword as a separate string and uses the AND operator, it should return between 0 (if none of the jobs contains both keywords) and 9 (if all "landscaper" jobs also contain the keyword "analyst") rows;
3. if the search treats multiple keywords as a single string, it should return between 0 and 9 rows (just like above).

The question is: Why the heck this thing returns 168, which is totally absurd?

But an even bigger question is: If it took me less than two minutes to break, why hadn't anyone done so before this piece of... software was put in production?

Yes, I did submit a bug report for the above, and I have not even received a confirmation. Well, I guess, some organizations can afford to not care how they are perceived by users.



Then, out of curiosity, I decided to have a look at the actual search results for "landscaper" (see screenshot 4 below this post). As you can see, 6 out of the total of 9 records returned by the search are not even close to landscaping. 66.6% of search results are completely irrelevant. For example, here is the full description of job #907051:

Enterprise Project Manager Job

Overview
Category: Regular
Job Title: Enterprise Project Manager Job
Experience: 5 Years
Education Level: Bachelor's Degree or Equivalent
Minimum Salary: Competitive Salary
Maximum Salary: Competitive Salary
Shift: Day
Workweek: Full-Time
Hours Per Week: 40.00
Duration: 151 + Days

Summary
Project Manager with PMP certification. Five to ten years of experience relating to numerous Data Center Consolidations and Relocations with a proven track record of success in assisting to reduce costs associated with multiple Data Centers and minimize requirements for physical space, equipment, software, power, air, labor and administrational controls. Experience with varied types of Enterprise Mainframes and Servers, Storage and Networks.

Skills
Project Management; ITIL; Management of technical support personnel, technical writers, System Administrators, DBA's; Scope Management; Requirements Management; Schedule Management; Financial Management; Quality management; Resource Management; Communications Management; Project Change Management; Risk Management; Procurement Management; Details migrations plan.

Job Description
Responsibilities will include creating plans, communications and coordination of activities to minimize system downtime during the migration/relocations. Will be responsible for the management of all basic elements of scope, cost and schedule. Works with Client Team on Risk Management Plan, Change Control, Resource Management, Resource Plan and Communication Plan. Makes recommendations to use more efficient methods; streamline processes, Virtualization, by Participating in assessments to discover duplicate software, over allocated hardware.

Other
Must be on-site 5 days per week with no expenses for traveling unless off of the site. Reports to the Project Management Delivery Organization for personnel activities and Engagement Manager responsible to deliver the projects the candidate is assigned to.

If this has anything to do with landscaping, please, be the first to cast a stone at me!

I will save you the questionable pleasure of reading the other five job descriptions. Believe me, they all are straight IT jobs. None of them contains any words or phrases that may have confused the search, e.g. "tree" (as in "directory tree") or "root" (as in "root folder"). Three of the six jobs in question are marked as "green jobs", but that still doesn't explain how the other three IT jobs ended up being landscaping jobs (it's inconsistent).

The only explanation I can come up with is... the grass... or whatever these people were smoking must have been really potent... And, by the way, searching for "grass" returns jobs like Software Engineer, Web Producer, CNA, Inpatient Services RN, CDL Truck Driver, Investment Management Paralegal, and a whole lot more like that in addition to just a handful of somewhat relevant results.


All this could be considered kind of amusing, but there is one problem: people who receive unemployment compensation from the state are actually required to use this... "system".



UPDATE:
To be fair, I have to mention that I received an e-mail from the support people on 7/27/12. It did not, however, address any specific technical issues and most likely was a canned message.

---

Screenshots (the same ZIP code was used in all searches; it is not shown on the screenshots to prevent the state from being identified; job locations and logos on screenshot 4 are not shown for the same reason):

1.

2.

3.

4.