A star system, also known as a solar system, a planetary system, or simply as a system, is the area around a star or a group of stars that contains all of the planets, moons, asteroids, nebulae, and other landmarks which are held in place by the star's gravity well. (TOS: "The Cage", "Elaan of Troyius", "The Doomsday Machine", "Patterns of Force"; ENT: "Broken Bow"; DS9: "Starship Down")
Star systems have a habitable zone where the inhabited planets and planets that can be terraformed are located. (TNG: "The Battle" display graphic) A star system's boundaries are marked by the area in which a star or celestial body's gravitational force affects its surrounding environment. The edge of these boundaries may contain an Oort cloud. (DS9: "The Maquis, Part II")
Using warp drive to travel from within a solar system to another point in the system was not recommended. Kira Nerys performed such a maneuver in 2373 in the Bajoran system, against the recommendation of Jadzia Dax. (Star Trek: The Motion Picture; DS9: "By Inferno's Light")
Background information Edit
In the shooting script of Star Trek: The Motion Picture, the film would have begun in deep space. After witnessing a supernova, the camera would move on searching for V'ger. In its search, the camera would have seen the "jeweled beauty" of star systems.