The data rate is 20 times higher than the best commonly used wi-fi standard.
As consumers become ever more hungry for high data rates, standard lower-frequency bands have become crowded.
The research, published in Electronics Letters, adds to the idea that this "T-ray" band could offer huge swathes of bandwidth for data transmission.
The band lies between the microwave and far-infrared regions of the spectrum, and is currently completely unregulated by telecommunications agencies.
Despite the name, the band informally makes use of frequencies from about 300 gigahertz (300GHz or about 60 times higher than the current highest wi-fi standard) to about 3THz, 10 times higher again.
It is used principally for imaging in research contexts, as terahertz waves penetrate many materials as effectively as X-rays but deposit far less energy and therefore cause less damage.
Until recently, the technology required both to generate and detect these "T-rays" has been too bulky, costly or power-hungry to offer a plausible alternative to existing devices tucked within smartphones or wi-fi routers.
That looks set to change; in November electronic component firm ROHM demonstrated a 1.5Gb/s (1.5 billion bits per second) transfer rate at a frequency of 300GHz.
Terahertz wi-fi would probably only work over ranges of about 10m, but could in theory support data rates up to 100Gb/s - close to 15 times higher than the next-generation 802.11ac wi-fi standard that is under development.
The new work, by researchers from the Tokyo Institute of Technology, demonstrated 3Gb/s transmission at 542GHz.
At the heart of the team's 1mm-square device is what is known as a resonant tunnelling diode, or RTD.
Tunnelling diodes have the unusual characteristic that the voltage they produce can sometimes go down as current is increased.
RTDs are designed such that this process makes the diode "resonate", which in the current work's design means it sprays out waves in the terahertz band.
The team is now working to improve their proof-of-principle device and extend its range deeper into the terahertz regime, as well as increasing its power output.