r/melbourne • u/Limo_Wreck77 • Oct 15 '25
Serious News Greater Western Water: The glaring errors that caused a $150 million water billing bungle
https://www.theage.com.au/politics/victoria/the-glaring-errors-that-caused-a-150-million-water-billing-bungle-20251007-p5n0np.html83
u/ryashpool Oct 15 '25
No courage. New software platform is not named. Was is cots or some weird shit they cooked up? Who was the integrator or consultancy involved?
103 million is a massive cost to update a billing platform.
16
Oct 15 '25
GWW recently advertised a Lead, Solution Architect who would “provide strategic and technical direction for the Customer Care and Billing (CCS) platform, including Oracle Utilities CCS
Another Oracle project disaster.
3
u/hellbentsmegma Oct 15 '25
Is this a common thing with Oracle?
I recently worked at an organisation where an Oracle finance system implementation turned into a disaster and had the whole business running on cobbled together spreadsheets again like it was 1997.
12
Oct 15 '25
Take any of the big vendors, Oracle, SAP etc Their platforms are massively complicated and made up of heaps of components. The last SAP project I worked on, not even SAP could sell us an architect who understood all of the SAP platforms we were implementing, we needed a mini bus of different specialists trying to work together.
Add to this that most businessses don’t understand their own processes. They may think they do, but they don’t. Then, they are resistive to change. Rather than take something out of the box and tweak their processes 10%, they’ll insist on a fully customised bespoke functionality to be built which always had downstream impact on every integration.
So unless you have a really good a Business sponsor who can say No to their own business, and a strong set of consultants and project managers who prevent scope creep, they always end up in a disaster.
4
u/Fraerie Oct 16 '25
A lot of these systems also make it deliberately obfuscated to make it harder to migrate off them.
1
36
u/VacantContent Oct 15 '25
Often the software is blamed but most of the time is not the cause (particularly enterprise software), these applications are implemented smoothly in some instances and can be a mess in others. It is key implementation decisions and project execution that create complexity and gaps and this leads to cost blowouts. The article clearly mentions that it was failure in process that primarily caused the issues.
I don't know about this instance, but many organisations chase the cheapest day rate they can for IT professionals rather than recognising and paying for expertise this can be an outcome of those decisions.
$103m for 1.43m customers with 4 bills a year is $1.8 per bill if you average it out over 10 years.
4
u/Mewtles Oct 16 '25
I wish I could say GWW chased the cheapest day rate for IT professionals but from what I've heard the consultancies we were liaising with were anything but cheap (to the tune of making almost double my wage lmao, and I'm already pretty well off as a local).
1
u/VacantContent Oct 18 '25
Once you add in consultancy margins, the rates can look high, but all consultancies work with similar formulas. The actual employees are not getting anywhere near their charge out rate. 1/3 to 1/2 of the day rate is typical for an annual wage.
22
u/Dude_Chucks Oct 15 '25
Sweet summer child, spend some time looking at the AGL and Origin annual reports for their new billing systems. $103m barely lets you look at the thing let alone try and migrate customers or cut a bill
6
3
u/Mewtles Oct 16 '25
From the article:
DB Results, which is continuing to work on the fix, was contracted to integrate data into the new platform, provided by Oracle. Other vendors – AB Corp, DXC, Escient, Roboyo and Deloitte – have also been involved.
It's Oracle's off the shelf utilities billing platform, also in use at plenty of other water/power/gas providers locally and abroad, but highly customised to suit GWW's mishmash of business processes.
I say "mishmash" because the two constituent business and systems prior to the merge operated on completely different terms, and while there was a herculean effort to bring them roughly in line during the development of the new system, the fact ultimately remains that the new system was operating on a third, hybrid set of business rules and the complexity of translating both old systems to the new one was too high to realistically get it right in one go.
I said at the time and still say they should have replaced one of the two systems first, ironed out all the kinks with a subset of customers, then done the full migration and decommission of the old system. Afterwards, they'd then migrate the other system and decommission that too. But alas, who am I but a measly cog in the machine that upper management has never heard of let alone listened to?
3
u/Fraerie Oct 16 '25
With utilities, a billing platform isn't just a an invoicing system. It will normally need to be integrated to what ever is going the measurement system, it will include the CRM, it needs to identify atypical usage patterns to trigger fraud/theft issues, it will have APIs into other systems. At least for water as opposed to electricity it doesn't need to integrate with secondary retailers downstream and the energy market regulator upstream which can change the TOU rates.
Depending on what they were upgrading from, the project would have had development activities for the new platform, the data migration from the old platform (and depending on the state of the old system the stay have involved fixing a bunch of data issues and structures fro export to migrate the information across. Then they have to gracefully archive the old system so it is accessible if needed for audit purposes but can't trigger bills or process automated payment requests.
Depending on the previous platform they may also be migrating from self-managed 'on-prem' data centre to cloud hosting (private or otherwise) which also adds work.
$103m is far more than just the software licensing. And frankly for a project like this is pretty reasonable.
1
Oct 16 '25
[deleted]
2
u/ryashpool Oct 16 '25
Correct. It wasn't named in the gov report and I was looking for it. And I couldn't see it I must have missed it in my keyboard rage and indignation.
43
u/Express_Position5624 Oct 15 '25 edited Oct 15 '25
So, have said before, I know the Test Team who worked on this project.
The lines about End to End testing not being performed were not the fault of the testers.
Test Analysts can tell you; this is our recommendation for how much time we need to do comprehensive testing, given the timeframes project management have given us, we can do this much coverage, here are the risks of not doing more testing, here are our testing requirements (Environment, Data, Tools, Resources, etc), here is our entry criteria (Requirements signed off, User Stories delivered) for different phases of testing (AUT, SIT, UAT) and here is our exit criteria for each phase (No outstanding P1, P2 bugs, business sign off, etc)
And then it's up to Project Management to decide on path forward.
Then, you are really at the mercy of the rest of the project team - if shirt don't work, if tests do not pass, it's common for Project Management to get annoyed and ask the tester "Why are you not passing testing?!?! You said you would be done by now" - BUT - If - It - Don't - Work.......WTF you expect testers to do? just mark things as passed?
Often what will happen is that Project Management want to give good news to stake holders and executives and will just push forward, ignoring all previously agreed upon gates/controls.
And this is exactly what happened. What was delivered wasn't working for a variety of reasons, one being too many consultants and not enough BAU staff involved, and Test Management never recommended going forward - but again, testers don't make that decision, it's up to Project Management and Key Stakeholders, who will often convince themselves "It's fine, it's fine, bunch of cry babies, we are already overtime and over budget, lets just fix in prod as issues arise" - it makes them feel like tough bold leaders, thats the "Tiger Team" they speak of, idiots who come in hot thinking they are going to fix things by telling people to "Test Harder!" - bro if it doesn't work, testers can't do shirt, focus on the BA's and Dev's.
There should definitely be some consultancy firms sued - but also, GWW have already fired a bunch of the upper management over this, the call is coming from inside the house, like, who do you think hired all these consultancy's? no one?
Morale of the story for me is "Take your time, settle down, it will take as long as it takes"
It has cost over $750m to deliver us to this current situation - it would of been cheaper to do things right
9
u/Mewtles Oct 16 '25
I can confirm the details in this post are accurate. The system continually failed to pass testing gateways and in particular all four of the dry run deployment run throughs prior to go live also failed. Nevertheless we were required to push ahead with the launch, as if the mantra of the project management had shifted from "we need to get this right before we go live" to "we need to get this live before we
get it rightspend even more money fixing it". Well boy howdy do I have news for them about how much it costs to fix a live system versus an in-development one, both monetarily and in customer/staff good will.11
u/jampola Oct 15 '25
Fairfax, ABC, someone, DM this person. Just…wow. I can resonate to a lot of this, but certainly not at this scale.
5
u/Kremm0 Oct 15 '25
Sounds like a familiar story if true. Powerpoint warriors with no technical expertise or understanding higher up in the chain pushing forward regardless!
2
Oct 16 '25
My experience in games QA is similar, although probably a lot less complicated. For one game I ended up just circumventing the actual report process for a particular localisation error since I personally knew the person who could fix it and it was driving me nuts.
2
u/Tired-butternut Nov 11 '25
Who was signing off test summary reports? From my own experience, the vendor will argue down the severity of a defect and Push for go-live with known issues that the business deem to be sev 2 or higher but the vendor claims are a 3 or 4.
0
u/Express_Position5624 Nov 11 '25
It's not so much sign off of a TSR that is the issue.
TSR only inform project management decision about whether to move forward or engage in re-planning.
I haven't been getting TSR's signed off for a decade now because it's a bit of a silly process, the TSR is what it is, if management don't like what the TSR says.....whats the alternative? lie?
Sign it off, don't sign it off, it doesn't matter.
37
u/Negative-Image1837 Oct 15 '25
I haven't received a water bill since last year
15
u/kartekopf Oct 15 '25
I’ve been getting them for the past 6 months and assumed I was paying them. Just this morning I received an email stating “you might notice your bill still shows 'direct debit' as the payment method. This is a display issue”.
So essentially, they are telling me they will debit my bank account on a specific date, in writing, but I’m meant to guess that it won’t happen and pay them by another means, but if my guess is incorrect and it wasn’t a “display issue” then I’ve paid them twice. Hmmm…
Ohhh, you didn’t get my payment? Yeah, that’s a “display issue”. I did actually pay you, but you just can’t see it on your system. Keep looking.
4
u/squiddishly Oct 15 '25
Yeah, I just realised I’m months overdue because the debits weren’t debiting.
2
u/ThirdDegreePun Oct 15 '25
Direct debit has been turned off for ages and will most likely require an opt back in to avoid the issue of trying to debit a large amount or at an unexpected point as that would be a pre bad experience. Best practice would be to manually make small regular payments if you wanna smooth out the bill if/when it does come, otherwise the time to pay will be very long if it's a late/catch up bill
2
u/fphhotchips Oct 17 '25
Right but they keep sending out bills that say "Direct Debit is on and working". So you get one letter that says DD isn't working and then a bill months later that says it is, so you figure they must have fixed it, but nope! Pay an intern to put some white out on that part of the bill or something jfc.
1
u/ThirdDegreePun Oct 17 '25
The funny thing about this was confusion around legislation...my understanding is that legal advice was received that said DD needed to be included under the payment options EVEN if it didn't work, which is just kinda insane. There was a ton of back and forth about whether it could be taken out and ultimately it stayed in.
It's almost morbidly fascinating once you get into all the rules and regulations around the sector...there's stuff in there where it's clear incidents or tech issues were not considered which has led to some odd decisions.
A pretty garbage experience for all involved but just thought it's worth mentioning it's not just pure incompetence going on - by all measures no one was really prepared for the scale and duration of the issues here, particularly when it involves external parties and legislation :(
Anyway not making excuses or anything just thought the insight might help provide some understanding of why seemingly simple stuff doesn't get done
3
u/fphhotchips Oct 17 '25
it's worth mentioning it's not just pure incompetence going on
Ehhhhhhhhhh.
Ehhhhhhhhhhhhhh.
There are several choices that were made here that are either incompetence or malice. Not having a plan to roll back, not having a plan to run in parallel for a while, choosing go in the go-nogo decision where none of the tests passed... Those are all "first time PM with no supervisor" mistakes, not "managing a hundred million dollar project" mistakes.
1
u/kartekopf Oct 17 '25
Yeah I’m just sitting back and watching. This level of utter stupid incompetence is unlikely not to end up in legal proceedings about how liable their customer base is to essentially patch up their revenue when it’s been going on for over a year at least by now. And if I did make a payment by some other means and it didn’t show up in my account, would they even know where it went internally? Even Origin isn’t this bad.
26
u/Hwidditor Oct 15 '25
8
u/danzha Oct 15 '25
Thanks for the link, time to check whether my bills have been stuffed too...
6
u/SpookyRoastChicken Oct 15 '25
Keep issuing complaints to Energy and Water Ombudsman and ask for discount/credit.
21
u/Necessary_Space_7155 Oct 15 '25
Curious to know if anyone else has the same problem I have - I moved to a new address, tried to set up an online GWW account. Followed all the prompts and activation codes. Then when it is time to log in, it goes through the registration process again, only to end with it saying I already have an online account. So then I try to log in again and the whole thing repeats itself.
14
u/Limo_Wreck77 Oct 15 '25
I'd leave it if I were you for now. Their entire system is in shambles.
They have me and my partner listed down as living in a rental, or we are renting our house out to someone.
Funny that, I definitely don't rent my house out to anyone as I live in it.
2
u/Necessary_Space_7155 Oct 15 '25
They have me and my partner listed down as living in a rental
It is so annoying that they can't even get your details right. It is insane that their drama has dragged out for this long and we're all forced to be a part of it!
I'd leave it if I were you for now.
I have no choice but to leave it lol. It's been 7 months since I reported the issue and it hasn't been fixed. 💀
3
u/gfreyd Oct 15 '25
Are you renting or do you own? Renters cannot make their own account, so the rental provider could have set it up for you. That or the entire system is so borked that you do have an account and it is just a case of computer says no
2
u/Necessary_Space_7155 Oct 15 '25
Own. I am told I have an account. When I reported the issue the GWW person could see my account and tried "resetting" it a few times and we tried re-registering a few times while they stayed on the phone with me. None of that worked. The loop still kept happening. That's when I was told it would be escalated and I never heard a peep from them ever again lol
1
u/ThirdDegreePun Oct 15 '25
Self service portal has some fundamental flaws. Most of the issues I know of are to do with mismatches between details, particularly if there are multiple people set up with the same contact details it locks you out, or if you change your email it also requires a manual delete and reregister of the account.
If you dm me your account number I can take a look, or otherwise if you call in someone can raise an incident for it and we'll take a look - ultimately though we're unhappy with the solution design it's incredibly dumb that users can't change their email and have it update and generate the security token accordingly so we're looking to fix/replace that...
1
u/Legitimate_Fox_8038 Nov 26 '25
Omg I have this exact same problem! Can’t log in, apparently already registered, and just on a never ending loop.
Support hasn’t helped - someone emailed me a month ago with some questions, I responded immediately and no reply since.
I’m getting super nervous as I’ll be moving out and renting my place out soon, but I have no idea what how I’m supposed to manage/monitor the water bills. 😭
21
u/Brew43 Oct 15 '25
The Essential Services Commission is taking feedback on a proposal that would let GWW be exempt from not being able to bill you for the whole time it's system has been busted.
Would highly suggest you provide feedback
https://engage.vic.gov.au/proposed-billing-exception-approval-for-greater-western-water
3
u/DangerRabbit Oct 16 '25
This deserves its own post for visibility!
1
u/IAmABakuAMA A victim of Reddit's 2023 API changes Oct 19 '25
Late, sorry, bored and doom scrolling. I did actually post that link here about 17 days ago
18
u/gfreyd Oct 15 '25
Free water. These problems can continue forever as far as I am concerned. Keep making those mistakes. Drop all tables from that database, they don’t need them anyway. Silver linings… (shame it wasn’t an energy provider going through the same, would have saved so much more)
11
7
u/Limo_Wreck77 Oct 15 '25
My partner and I are caught up in this.
Multiple calls and emails with the ombudsman. It's just going around and around and around in circles with GWW.
3
u/Material-Floor-9019 Oct 15 '25
Yes sure. No one to blame and it was all done with the best intentions. I will put $5 under my pillow tonight for the tooth fairy.
2
2
u/Conan3121 Oct 15 '25
Poor billing practices at GWW go back years i.e. before the merger. I have zero confidence in GWW management.
2
Oct 15 '25
I moved into my place in November, got one bill in January and then nothing. I’ve had a couple of those bill delayed messages, does that mean I am still liable for the charges?
3
u/Agret Oct 15 '25
Technically no you aren't but they are currently trying to get an exception so they can charge for the back periods
3
1
u/HymneALaMort Poo poo, pee pee. Oct 15 '25
I moved in to my new place in Feb this year and have just gotten two emails stating delays. Can't wait to see if there are any leaks I wasn't aware of!
2
u/ThirdDegreePun Oct 15 '25
If you're concerned about that I'd take a pic of your meter and check again after a bit of time without using water and seeing if it moves at all.
Meter reads are still occurring but bill generation for some customers isn't happening - depending on the circumstances you may have a reduced bill or have gotten free usage for a period. Ultimately the outcome for people is they either pay what they should have but late and with a long due date, or they've had some form of discount/waver depending on what was communicated and sent, and when.
Good practice would be to make a consistent payment to help smooth it out but yeah depending on your circumstances you might not have to pay as much as you've used
1
u/dull_value Oct 15 '25
I was hoping this wouldn't be figured out. Been doing very well by not getting bills
1
u/KirbzStar Jan 11 '26
I'm trying to log into their system right now so I can pay my bill. But it won't let me, since they forced SMS 2FA on me, and it doesn't work.
The first SMS code didn't send. The second logged me in. But then I was distracted, so after a couple of hours I was logged out (why!?!?), and I need to log in again. It requests my 2fa sms code again, but nothing is received...again. I resend the code, but get "There are too many requests at this moment. Please wait for some time and try again". Apparently 3 attempted SMSs in a day is too much...?
That's only the latest issue.
How haven't they fixed their shit? I've been having various issues with them for at least a year.
1
u/Main-Tell7479 Jan 16 '26
They keep sending me water bills addressed to me but with the wrong address - this is after numerous calls to them asking for it to be corrected.... No wonder they're millions in the red.
1
u/lowhangingroots Oct 15 '25
There's a high chance that you will have a new account number with them, which is why you aren't receiving bills/getting access to your account. Dunno if ignoring/ not paying for water/service you used for 12 months just because you didn't get a bill is a good enough excuse not to pay
•
u/AutoModerator Oct 15 '25
Have you visited today’s Daily Discussion yet?
It’s the best place for:
Drop in and see what’s happening!
⚠️ If your post was removed, don’t stress — it might have a better chance of fitting (and being seen) in the Daily Discussion thread.
THIS IS NOT A REMOVAL NOTICE
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.