Here's a little info on the attacks that did occur to Australia during 1942-45 courtesy of Emperor Hirohito and his Service Leaders:
Attachment 107405
One of the chilling attacks that shook contemporary Australia to the bone was the Jap midget subs that attacked inside Sydney Harbour and sank the former Sydney Ferry HMAS Kuttabul, for the loss of 19 RAN and RN sailors on board. Darwin's repeated attacks were less of a 'concern' as the communications in those days isolated the population. But the Sydney attack was a real wake up call. We got one of the subs, sans crew, and the remains are displayed at the AustralianWar Memorial in Canberra. Brave Jap sailors who came in to terrorise us, they did thier job well.