Seaweed worms belong to the phylum Annelida subclass oligocheata. I suspect the term "garden worm" is used to denote the subclass oligocheata and differentiate them from the polychaete subgroup (tube worms, beach worms, blood worms etc).
"Aquatic oligochaetes are closely related, and quite similar, to earthworms. There are over 3100 species of terrestrial, marine and freshwater oligochaetes worldwide. Ten families of freshwater oligochaetes occur in Australia, represented by over 90 species. In South Australia, there are at least 35 known species."
https://www.ep