Accurate 3-D modeling of an underground structure is one the most important factor for precise simulation of seismic ground motion. Modeling of the basement structure using travel-time data only sometimes encounter the ray covarage problem. Certain parts of the model cannot be reached by rays. We develop 3-D joint inversion of refraction and gravity data to solve this problem. Beside that, we have improved finite difference calculation of seismic travel-times. The method is applied to the data obtained around Osaka Basin, Japan. The result agrees well with geological setting in this region. The hidden Koyo fault and the extension of Ashiya fault are recovered clearly. The velocity and density of the basement are 5.341 km/s and 2.384 g/cm^3, respectively, which represent the elastic characteristic of the weathered granitic rocks which include ryolite and metamorphic rocks in this area. The instability due to the lack of data beneath Osaka Basin is improved by introducing depth constraints.