NYCPHP Meetup

[David Krings]

David Mintz wrote:
> I get the impression that most developers report that they do automated 
> testing or they are trying to learn. Only a minority seem to be 
> answering flat-out no.

Automated testing is the most overhyped tool in the QA world. I am in software 
QA for a decade now and automated QA makes sense for regression and load 
testing, but not for any new development. Test scripts are nothing else than 
new code, which again needs to be tested.
Ideally, the developer writes the test script for the code he or she is 
working on. Both application code and test script are only as good as the 
functional requirements and technical specifications. The better the 
expectations are definied (including what the feature is supposed to do and 
what it is NOT supposed to do, which is the part most commonly omitted), the 
better the outcome is. Crappy specs make crappy code and no matter how much 
automated testing is thrown against it will uncrappify the code. There is also 
often a general disconnect between developers and end-users, but that is 
fostered by the unsuitable management structure of most software companies. 
Support needs to be at the top with the biggest weight, below that are product 
management, then QA, and development is really at the bottom. I know it will 
come across as provocative, but developers are overrated and overpaid. They 
tend to get the nice cubes, the latest technology, and if the project plan is 
done right have a 9 to 5 job. Support, QA, and thus the customer have to 
suffer from whatever the devs crank out. Automated testing doesn´t do a thing 
for that.
BUT, devs can put themselves, their coworkers, and the customers in a position 
where nobody has to deal with crappy code. Follow coding guidelines, code 
modular, reuse known good code (which is only a small fraction of all code), 
and document the hell out of it. The next guy or yourself six months later 
will be happy. And always keep the end-user in mind. Way too many technology 
decisions are made so that the developers have it easier to code stuff, even 
if that means that implementing and maintaining it will be more difficult. So 
code in a way that verbose logs can be pulled upon request, that the data is 
easily transitioned, that apps are cross-platform, and then installation and 
maintenance are as easy as possible. And those are things where being smart 
helps way more than some automated test suite.


> imho one of the major differences between a polished professional and a 
> hobbyist is responsible testing. That's why I still kick myself, as sort 

That has nothing to do with professional or hobbyist, but with good and bad 
developers. Good developers make very sure that anything they hand out is 
definitely going to work. And if it fails they put the means in to find out why.


> of an eternal wannabe whose day-job description has zero to do with IT, 
> but who aspires nevertheless to write code that can be judged competent 
> by any standard:  I still haven't gotten the hang of unit testing, 
> though I have made some attempts to get religion and vow to keep trying.

Unit testing is really a bad name, since it is not part of the testing task, 
but part of coding. I have seen plenty of code that all it did was compile. It 
was obvious that the dev didn´t even try it out once. Where I work we have the 
devs detail in the tech specs which things need to be tested and how. That 
isnät in as much detail as the test plans, but it makes the developers try 
things out first. The task of a developer is way broader than just writing code.


> It seems that I am not alone. So the question is, why is it hard for 
> some to adopt the practice? One thing that occurs to me is that maybe it 
> isn't really that easy to write good tests. We've all seen examples of 
> how to test a function that returns the sum of x + y -- thanks a lot. 

Testing and with that bug fixing is a tedious, frustrating, time consuming 
task that adds no new functionality to the application. Fixing bugs is 
probably the least favourite things for a developer to do, comes right after 
writing decent documentation (and tech writing should be a mandatory class in 
CS programs). It is less annoying when testability, fixability, and 
maintainability is designed and coded into an app right from the start. This 
is why it is so utterly important to design quality into the app way at the 
beginning. Unfortunately, it is often only an afterthought. See, I dabble once 
in a while in PHP and fixing things is really not as much fun as writing new 
stuff. So when writing new stuff taking effort to make the bug fixing to be a 
non-issue will make everyone happy.

> It's a little harder to create simulated http environments complete with 
> cookies and sessions -- though people (e.g., the ZF team) have done 

Why simulate? You can grab about any PC and make it a (miniature) version of 
the production environment. Sure, if you decide to use for-pay only platforms 
it might cost you. And with ESXi being free to download and refurbished Dell 
servers selling for a few hundred bucks setting up VMs for testing is almost 
an option for the hobbyist. And PHP really helps a lot here compared to 
ASP.NET for example. Sure, VisualStudio is an awesome IDE, but you and your 
users will pay dearly for it for time to come. This shows how much rides on 
technology decisions, because they determine how the end product pans out even 
before a single line of code was written.

> wonderful things to make it easier -- and forsee all the use cases in a 
> complex app. Evidence for the proposition that it's hard to write an 
> exhaustive set of tests is that you have products like the Zend 
> Framework which is unit-tested to death before release, and then the 
> next release brags that 171 bugs were fixed.

The QA analyst speaking here: you cannot test everything. As soon as you allow 
for combinations of things and free form user entries you can be very sure 
that someone out there will make your app blow up. And I don´t think that is a 
problem per sé, the problem comes when the bug isn´t dealt with the right way. 
There are only a handful of programs that are generally considered reasonably 
bug free. So if the Zend folks fix 171 bugs and got it right the second time 
around, then it for sure is something to brag about. I´d stop celebrating when 
the 171 bug fixes caused 271 other things to break or worse now break 371 
applications based on that platform. Still, it also gives 171 very good clues 
where the QA process failed and should result in at least 171 more test cases 
added to the test plans.


And if you think my emails are long you should see the test plans that I 
write.... ;)

David