Answer :
Let A(t) denote the amount of salt in the tank at time t.
Salt flows in at a rate of
(1 lb/gal) * (3 gal/min) = 3 lb/min
and flows out at a rate of
(A(t)/(200 + t) lb/gal) * (2 gal/min) = 2 A(t)/(500 + t)
(in case you're unsure about the denominator: the tank starts off with 200 gal of solution, and each minute solution flows in at a rate of 3 gal/min and thus the tank gains (3 gal/min) * (1 min) = 3 gal. At the same time, solution flows out at a rate of 2 gal/min and thus the tank loses 2 gal, giving a net change in volume of (3 - 2)*t = t gal)
Then the net rate of salt flow is given by the ODE,
[tex]\dfrac{\mathrm dA(t)}{\mathrm dt}-\dfrac{2A(t)}{200+t}=3[/tex]
Multiply both sides by [tex](200+t)^{-2}[/tex]:
[tex](200+t)^{-1}\dfrac{\mathrm dA(t)}{\mathrm dt}-2(200+t)^{-3}A()=3(200+t)^{-2}[/tex]
[tex]\implies\dfrac{\mathrm d}{\mathrm dt}\bigg((200+t)^{-2}A(t)\bigg)=3(200+t)^{-2}[/tex]
Integrating both sides and solving for [tex]A(t)[/tex] gives
[tex](200+t)^{-2}A(t)=-\dfrac3{200+t}+C[/tex]
[tex]A(t)=-2(200+t)+C(200+t)^2[/tex]
The tank starts off with 100 lb of salt in solution, so [tex]A(0)=100[/tex] and we find
[tex]100=-2(200)+C(200)^2\implies C=\dfrac1{80}[/tex]
and so
[tex]A(t)=-2(200+t)+\dfrac{(200+t)^2}{80}=\dfrac{(200+t)(40+t)}{80}[/tex]
The tank will begin to overflow once the volume of solution reaches 500 gal; this happens when
[tex]500=200+t\implies t=300[/tex]
or 300 minutes or 5 hours after solution starts flowing. At this point, the tank will contain
[tex]A(300)=2125[/tex]
or 2125 lb of salt.
Theoretically, the amount of salt in the tank will increase forever, since [tex]A(t)\to\infty[/tex] as [tex]t\to\infty[/tex].