The pilot reported the agriculture airplane was loaded with about 5,000 pounds of fertilizer and ¾ tank of fuel; he planned to depart a private grass airstrip which was about 2,600 feet long. The pilot departed the runway to the north, and the airplane's landing gear impacted a levee at the end of the runway. Once the landing gear impacted the levee, the airplane would not gain altitude and the pilot conducted a forced landing in an adjacent field. Examination of the airplane revealed substantial damage to the fuselage during the landing. The pilot reported there were no preimpact abnormalities with the airplane and added that a gust of wind or whirlwind caused the airplane not to climb. About the time of the accident, the temperature was 87 degrees F with a southern wind about 5-10 knots. It's likely the downwind departure, degraded the loaded airplane's takeoff performance.